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ABSTRACT 

The Agricultural Research Service (ARS) is the 
primary research agency of the U,S, Department of Agriculture, ARS 
scientists conduct research into the applications of technology and 
basic knowledge concerning food and agricultural enterprises. Area 
offices and national staff evaluate these scientists' proposals for 
in-house research through a system of peer review. An ARS committee 
was appointed to examine the peer review system and to recommend 
possible improvements. This report presents the committee's findings 
and recommendations as well as background information to serve as an 
introduction. Appendices are listed under the following headings: (1) 
"Research Project Peer Review Form (ARS-415)"; (2) "The Six Major 
Objectives of the ARS Program Plan"; (3) "Other ARS Review Systems"; 
(4) "In-house Review in Other Federal Laboratories"; and (5) 
"In-house Review in Private liidustry," (CW) 
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Preface 



The Agricultural Research Service (ARS) is the principal re- 
search agency of the U.S. Department of Agriculture (USDA). 
ARS scientists conduct research to increase basic knowledge and 
apply modern technologies to improve the nation's food and agri- 
cultural enterprises. This important research mission requires a 
dedicated staff working within a coordinated framework. The ARS 
carries out its objectives in its widespread network of laboratories 
staffed by scientists trained in the many disciplines that support 
agricultural research. 

The laboratories are guided in their unique mission by congres- 
sional directives, the ARS's centralized National Program Staff, 
and the area offices that manage the ARS's geographic groups of 
research locations. Area offices and the National Program Staff 
evaluate their scientists' proposals for in-house research projects 
through a project peer review system. Peer scientists inside and 
outside the ARS analyze written project proposals for their scien- 
tific value, methodology, and mission relevance. The goal of this 
review system is to help raise the quality of ARS research. 

The ARS administrator asked the Board on Agriculture of 
the National Research Council (NRC) to examine the project peer 
review system, assess its effectiveness, and recommend possible 
improvements. In response to this request, the NRC established 



a committee that was knowledgeable and experienced regarding 
peer review, the conduct and management of research, and the 
ARS. 

The conmriittee met twice and spent ccmsiderable time in- 
terviewing ARS staff members. The conmiittee received excellent 
cooperation from ARS staff members and was impressed with their 
presentations. In arriving at its findings and reconmiendations the 
conmiittee relied mainly on the expertise of its members and on 
the interviews in addition to its review of a large volume of data, 
information, and statistics. 

The conmiittee's findings and recommendations, which form 
the substance of this report, are put forth to help the ARS realize 
the fullest possible benefits from peer review to its research efforts. 
The committee has also included background material on the ARS 
and peer review as an introduction. In addition, the appendixes 
describe selected peer review systems used to evaluate personnel 
and programs within tl ARS, other government agencies, and 
private industry. 

RICHARD S. NICHOLSON 
Chairman 
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Introduction 



THE AGRICULTURAL RESEARCH SERVICE 



ARS Mission and Obj<»ctiyes 

The Agricultural Research Service (ARS) is the principal in- 
house scientific research agency of the U.S. Department of Agri- 
culture (USDA). ARS scientists conduct R&D on food and agri- 
cultural problems of broad scope and high national priority. ARS 
research is primarily mission- and problem-oriented. ARS pro- 
grams include basic research as well as applied R&D. The congres- 
sional appropriation to the ARS for fiscal year 1986 was $509.7 
million, much of it earmarked for research on specific commodities 
and agricultural problems. ARS research must be consistent with 
mandates established by Congress and responsive to the needs 
of the USDA, other government agencies, private trade organi- 
zations, and other users. Tne six major objectives of the ARS 
program plan are listed in Appendix B (ARS, 1983). 
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ARS Organiiation 

The administraijr takes ultimate responsibility for ARS pro- 
grams. The National Program StaF i? responsible for overall pro- 
gram direction and allocation of funds for specific research pro- 
grams and proje-^ts. The National Program StaflF includes about 31 
senior scientists representing many fields of agricultural research. 
These national program leaders report to two associate deputy 
administrators and the National Program Staff deputy adminis- 
trator for programs. These administrators determine fund distri- 
butions within and among commodities, program plan categories, 
and research projects subject to the constraints of congressional 
mandates. The administrators' approval is needed for research 
projects. 

The ARS's 127 research locations are geographically distrib- 
uted among eight area offices: North Atlantic, Beltsville, South 
Atlantic, Midsouth, Midwest, Southern Plains, Northern Plains, 
and Pacif c West. An area director is in charge of each location. 
Among other duties, area directors are responsible for conducting 
the project peer review process within their areas. There are ap- 
proximately 1,600 in-house, appropriated ARS research projects; 
20 to 25 percent of these are completed each year. Thus, normal 
project turnover provides opportunities for redirection of research. 

The ARS employs more than 8,500 people full-time. Scien- 
tists and engineers represent about 35 percent of this work force; 
research technicians and support staff, about 65 percent. The staff 
are customarily granted permanent appointments or tenare after 
one year of service. The attrition rate of scientists is 3 to 5 percent 
annually. The ARS management therefore has the opportunity to 
fill about 100 poets each year with scientists skilled in new areas 
important to the ARS mission. 

A senior investigator or research leader directs each research 
project. Each research leader prepares an annual resource manage- 
ment plan that must be approved by his or her area director, the 
National Program Staff, and the ARS administrator. The National 
Pro{^ram Staff, leading scientists, and other experts continually 
assess national research priorities to ensure national coordination 
and program balance (ARS, 1985a). 

The National Program Staff maintains a computerized ARS 
Research Project System, which works with the Cooperative State 
Research Service's Current Research Information System (CRIS). 
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These aystems constitute a ce^ alized on-line information source 
documenting agricultural and forestry research projects in the U.S. 
Department of Agriculture, state agricultural experiment stations, 
and schools of forestry and veterinary medicine. An important use 
of CRIS by the ARS is to track fund allocations among research 
projects. These research protects are referred to as CRIS work 
units. 



PEER REVIEW IN THE ARS 

Roles of Peer Review 

Peer review of science ii the evaluation of the conceptual and 
technical soundness of rebearch by those qualified to judge it by 
their status in the same or clorely related research fields. Sci- 
entific peer review originated in the evaluation and approval of 
manuscript*j before their publication (Garfield, 1986; Zuckerman 
and Merton, 1971). The practice of reviewing manuscripts wa^ 
instituted to preserve the credibility of scientists and their insti- 
'iirons and ensure jhe quality of published literature. 

Peer review was subsequently adopted to assess grant propos- 
als, scicitific programs, and scientists. This evaluative meoianism 
is based on the premise that scientific peers, by virtue of their 
knowledge and experience, are be %ble to critically examine pro- 
posed or completed research projects and give scientific opinions 
concerning the projects' merit, significance, and feasibility. 

Federal granting agencies that make extramural awards closely 
couple peer review of research proposals to allocation of research 
funds. These agencies include the National Science Foundation 
(NSF), the National Institutes of Health (NIH), and the USDA 
Competitive Research Grants Office. The exact mechanics of the 
review and decision-making processes vary somewhat in these and 
other such agencies, but their intent is the same — to allocate 
resources fairly in support of high-quality science in relevant fields. 

These granting systems are competitive. Peer reviewers judge 
the scientific merit of proposed research projects and usually con- 
sider \dditional factors such as the scientist's past performance and 
the personnel and resources available to the laboratory. Reviewers 
generally assign a priority score indicative of their assessment of 
the proposal relative to competing proposals. Agency program 
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directors use the reviews to judge whether or not the proposal 
merits the agency's support. 

Peer review in granting agencies is largely prospective. That 
is, proposed research projects are prejudged on their likely scien- 
tific and technical merit, importance, and success. Reviewers do 
consider retrospective aspects, however, such as the quality and 
quantity of the investigator's previous scientific output. In con- 
trast, other peer review systems are primarily retrospective. Such 
systems include personnel evaluations for promotion and tenure. 

Considerations other than scientific excellence may enter into 
review processes. Such processes are collectively called merit re- 
view to indicate that other factors carry some weight. These 
factors may include nontechnicrJ policy considerations. One con- 
sideration may be the utility and relevance of research to a goal 
extrinsic to the research project itself, such as new or improved 
technology development or the solution of social problems. An- 
other may be the impact on the infrastructure of science such as 
quality, distribution, or effectiveness of research, education, and 
manpower. Additional factors may sometimes pertain, such as 
the relevance to mission-oriented goals of a sponsor, re^arch site 
selection, or interdisciplinary character of some areab of science 
(NSF, 1986). 

The role of peer review is most predominant in assessing re- 
search proposals from single principal investigators. These projects 
are known as ''small science.** In contrast, large, complex, ex- 
pensive, ^big science"* projects, such as the National Center for 
Atmospheric Research or the Fermi Accelerator, must pass the 
hurdles of technical scrutiny (peer review) and societal consider- 
ations (merit review). Most ARS research projects are conducted 
by single principal investigators managing small research teams, 
which is also common in universities. Tn this context ARS carries 
out small science. Therefore, technical peer review is sufficient for 
ARS projects. Before it appropriates funds. Congress has already 
considered the societal aspects of ARS scientific research. 

Federal agencies review their in-house or intramural research 
programs by various mechanisms. Outside advisory groups review 
NIH intramural programs by conducting retrospective evaluations 
of individual laboratories and their scientists. The advisory groups 
transmit formal reports to NIH top management. These reports 
influence promotion, tenure, and resource allocation. The Depart- 
ment of Energy (DOE) National Laboratories assess programs and 
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projects by severid prospective and retrospective methods. Private 
industry uses various types of review for planning, assessment of 
progress^ and personnel evaluation. Information on several federal 
agency and industry internal review systems is included in Ap- 
pendixes D and E. A key feature of many of these systems is a 
direct linkage to resource allocation. 

Erahiatbe Role of Peer Review Within ARS 

The ARS uses peer review in several ways: to examine the 
quality of specific research projects, assess its national research 
programs, and evaluate its research personnel. The ARS project 
peer review system is outlined in the next section. The ARS review 
systems for programs and personnel are described in Appendix C. 
These three review systems are not directly linked to each other. 

Project peer reviews do not directly influence whether or at 
what levels ARS research projects are funded. The National Pro- 
gram Staff's stated goal for the ARS project peer review system 
is to improve the quality of research already requested and funded 
by the federal government. The noncompetitive funding process 
of ARS laboratories differs from that of many university laborato- 
ries, which must obtain the majority of their research funds from 
competitive, peer-reviewed government granting programs. In ad- 
dition, ARS science is ission-oriented and conducted mainly by 
tenured government scientists. Nevertheless, project peer review 
is quite applicable to the continuum of basic, developmental, and 
applied research that the ARS conducts. Moreover, peer review of 
in-house government-funded research can be effectively integrated 
with processes of resource allocation, as in the NIH intramural 
research program reviews (see Appendix D). 

ARS Peer Review System for Li-Honse Projects 

ARS review procedures for in-house projects are linked to 
CRIS documentation, which tracks all ARS research projects 
(ARS, 1985b). Thus, project statements that ARS scientists pre- 
pare for review are related to standard forms they submit to the 
National Program Staff for updating its computerized ARS Re- 
search Project System. Ongoing projects must have their project 
statements rewritten and rert «riewed every five years. 

The project review process begins with discussion among the 
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Bcientiflt, hia or her research leader, area director, and national 
program leaanr. The discusmon leads to an informal understanding 
on the proposed content of the research project. In the case of 
newly appropriated as opposed to continuing funds, the associate 
deputy administrator writes a letter of instruction to the research 
leader identifying funds and objectives. Projects must fit within 
the problem statements of the ARS National Program Plan (ARS, 
1985a). 

The scientist prepares a detailed project statement for peer 
review. There is no prescribed format. Individual statement* 
vary greatly in length, from three-page summaries to fifteen-page 
(or longer) project proposals. The longer proposals may be mod- 
eled on formats that competitive granting agencies such as the 
NSF and the USDA Competitive Research Grants Office use. The 
project statement generally covers the project's objectives, justifi- 
cation, research approach, and a literature review. Details of the 
project's resources, personnel, and budget are not required; only 
its total annual budget is indicated. Descriptions of the scientist's 
past research accomplishments and publications are not routinely 
included. 

The scientist submits the project statement and a list of six 
suggested reviewers to the area office. The area office selects three 
or more reviewers from this list and from other sources. Reviewers 
may be from inside or outside the ARS. The area office mails the 
project statement and the ARS research project peer review form 
(see Appendix A) to its selected reviewers. The area office eval- 
uates the completed reviews and forwards them to the scientist. 
The area office may or may not forward reviews anonymously. The 
scientist must modify the project statement in accordance with the 
area office evaluation or otherwise respond to the reviewers' com- 
ments with a written statement. 

The scientist fills out /iRS research project sunmnary forms 
(AD-416 and AD-417: ARS, 1985b). The ftrea director must ap- 
prove the final proje^lr ?ia^ement and sumnuiry forms. Only the 
summary forms are fo^ « ltd ' the National Program Staff for 
review, approval, a . ci^ ^ • into *he ORIS computer. The scien- 
tist submits brief ennaaj trpor^^s to the National Program 
Staff (ORIS fom. i985b). 

The ARS prnjp : i^view system operates entirely prospec- 
tively. Site yiaii'^ ar r uot conducted in conjunction with these 
project reviews, ( ilie National Program Staff does periodically 
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visit ARS locations, however. See Appendix C.) Project reviews 
do not bear on the distribution of funds to ARS research projects, 
nor directly affect hiring, promotion, or tenure of ARS scientists. 
(ARS scientists are reviewed by their peers through an entirely 
separate Research Position Evaluation System; see Appendix C.) 
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THE COMMITTEES VIEW 

The committee notes that peer review is not a precise term; it 
means different things in different contexts. For example, in their 
extramural research grants programs, the NIH and the NSF use 
peer review as the primary method to assess the scientific merit 
of competing research proposals. In this context, peer review is 
critical in deciding which proposals will or will not be funded. On 
the other hand, the ARS states that it uses peer review to improve 
the quality of research projects that it is funding or intends to fund 
to meet its mandated responsibilities. The conmiittee supports 
this goal and the use of peer review to achieve it. The committee 
believes, however, that the ARS project peer review system needs 
to be strengthened to be effective. 

The ARS should be aware of the different meanings of peer 
review. It should articulate the goals of its projec^^ peer review 
system and the uses of the reviews. The project peer review 
system is only one of several systems the ARS uses to emphasize 
excellence in achieving its mission. In principle, all of these are 
complementary, but in practice they do not seem to be. The 
committee believes it is essential that the ARS treat project peer 
review as one very important part of the larger system. 

8 
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PRINCIPAL CONCLUSIONS 



The committee finds a lack of understanding and agreement 
among ARB staff regarding the purpose, use, and effect of the 
project peer review system. Many staff members also believe the 
system has no real impact on ARS research. As a result, some view 
it more as busy work than a substantive review of real or potential 
value to ARS scientists and, ultimately, to the ARS. This view 
seems logical because it appears the results of project peer review 
have no direct bearing on decisions affecting project funding, staff 
promotion, and merit pay increases. 

Moreover, there is inadequate understanding within the ARS 
regarding how the ARS administrator balances and optimizes the 
dual objectives of scientific excellence and mission relevance and 
how project peer review is used in the context of these objectives. 
The committee believes the National Program Staff, area directors, 
research leaders, and research scientists need to understand and 
conmiunicate better about the goals of project peer review, its 
implementation, and its impact on ARS science and scientists. 

The conunittee also perceives a great need to have more vis- 
ible incentives in the project peer review system. The staff must 
perceive that they will receive tangible rewards from full partici- 
pation in the system and from exemplary peer reviews. This leads 
to the conmiittee's principal reconmiendation. 

The ARS administrator should strengAen and reinforce the cred- 
ibility and relevance of the project peer review system by linking 
the system's outcomes to incentives and disincentives that all ARS 
staff recognize and understand. 

In its discussions the committee identified several mechanisms 
to implement this reconmiendation: 

• Better conmiunication by the administrator of the incen- 
tives and disincentives that already exist; 

• Coupling of project peer review results with budget allo- 
cations to research scientists and progranmiatic decisions of the 
National Program Staff; 

• Integration of peer review results with the ARS Research 
Position Evaluation System and with decisions related to merit 
pay increases and bonuses; 

• Correlation of exemplary peer reviews with allocation of 
resources, such as special equipment funds, increased staffing, 
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discretionary funds, travel, sabbaticals, and the ARS Postdoctoral 
Research Aasociate Program; and 

• Allocation of some research funds in identified mission ar- 
eas through eompetition based on the outcome of peer review. 
This necessarily implies that not all competitors will receive such 
funds. 

The committee does not regard this as an exhaustive list and 
urges ARS top management to identify additional mechanisms. 

ADDITIONAI FINDINGS AND RECOMMENDATIONS 

Managemcait 

The re$eareh management role of area directors should be strength- 
ened. They should be responsible for the success of their researchers, 
and they should identify and reward top performers. The commit- 
tee endorses the current discretionary fund for this purpose, but 
believes the size of this fund is inadequate. In addition, area direc- 
tors should have control over some of the mechanisms recommended 
above. 

Area directors and research leaders occupy key positions in 
determining the success of the peer review system. Success in 
scientific research depends greatly on the research environment 
and thus on the skill and wisdom of these managers. It is essential, 
therefore, that these supervisors have the authority, respoixdibility, 
and tools to manage the scientific excellence and productivity of 
their staff. 

Selection and Evalnation of Peer Reriewers 

The ARS should develop a system to permit identification and use 
of reviewers in addition to those selected by the investigator. A 
reasonable number of these reviewers should be from outside the 
ARS. 

Currently, an ARS scientist provides a list of peer reviewers 
for his or her proposal. The area director's staff makes the final 
selection from the list. This practice is contrary to that practiced 
elsewhere. It is a potential source of criticism and bias that could 
undermine the credibility of the project peer review system. 
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The ARS $hould implement additional proeeduree, aueh as the use 
of advisory committees, to assist in identification of appropriate 
proposal reviewers and conduct periodic oversight reviews. 

Related to this issue is the fact that the technical breadth of 
ARS projects makes it difficult for the small staff that manages 
the system at the area director level to make informed scien- 
tific judgments in many cases. The conmiittee believes there are 
other ways of conducting peer review that might overcome these 
problems. For example, the area director could have an advisory 
committee whose members would be consulted for names of ap- 
propriate revie^vers. In addition, such a committee could conduct 
periodic oversight reviews of the entire process. The committee 
believes, however, that it may be difficult to institute such a migor 
change for all of the ARS. It may be desirable to have one of the 
areas experiment wiLh this alternative at firsi.. 

Uniformity 

A policy and procedure manual should be written to standardize 
operations and accountability. 

Because judgment is involved in peer review, some flexibility 
and diversity are desirable. There should be a standard set of pro- 
cedures and recordkeeping in all areas, however. The committee 
also believes that uniform deadlines for the submission of project 
statements should be established throughout the ARS. 

Project Statements 

The ARS should provide a set of guidelines for preparation of 
project statements. These would include instructions about the 
length, scope, content, and organization of the proposals. 

There is great variability in the amount of detail and format 
of project statements. In some cases, insufficient documentation 
may preclude adequate peer review. 

Performance 

All project statements should include a record of the investigator's 
recent publications, past progress, and other accomplishments. Peer 
reviewers should be asked to comment on the likelihood that the 
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proposed research could be carried out successfully considwng the 
scientist's past and current performance. 

When a scientific research proposal is judged, the past and 
present performance of the scientist who submitted the proposal 
can be as decisive a factor as the proposed research itself. Such 
information is not currently included as part of the proposal and 
not considered by the reviewers. The committee believes that it 
should be. 



The ARS should determine and clarify its policy regarding reviewer 
anonymity. 

There is some misunderstanding and disagreement among 
ARS staff regarding the anonymity of peer reviewers. Most peer 
review systems protect reviewers' anonymity; this protection pre- 
sumably increases candor. The conmiittee supports reviewer ano- 
nymity or a modified procedure whereby reviewers are given the 
option of anonymity. 



The research project peer review form (ARS'^IS) should be revised 
and replaced with a much less structured one. Reviewers should 
only be asked to comment on the merit of the proposed research, 
whether it duplicates other work, and the likelihood that the inves- 
tigator could conduct it successfully considering his or her recent 
accomplishments. Reviewers also should be asked to recommend 
improvemenls in the research. 

ARS staff are generally dissatisfied with the reviewer form 
(ARS-415) because it is highly structured and asks for often irrel- 
evant information. (See Appendix A.) 



It should be clearly explained to peer reviewers that their role is to 
review proposed research on a problem that the ARS has already 
sdected as essential to meet its mission. 

ARS staff often do not fully understand the ARS's dual ob- 
jectives of scientific excellence and mission relevance and how peer 
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review is used in the context of these objectives to promote excel- 
lence in research projects. Reviewers unfamiliar with the ARS are 
even more likely to lack appreciation of the ARS's goal to balance 
and optimize these objectives. 



RELATED OBSERVATIONS AND RECOMMENDATIONS 

Laboratory Site Visits 

The ARS should increase its use of laboratory site visits as a 
mechanism for reviewing the scientific merit of ongoing research 
projects and should involve ARS staff and outside peer reviewers as 
part of the site visit team. 

The project peer review system operates in the context of 
several other review systems. One of these is periodic site visit 
reviews of laboratories by ARS staflF, including the National Pro- 
gram Staff. The conmiittee endorses this process but believes that 
it would be enhanced if the site visit teams included outside pe^jr 
reviewers. Other agencies such as the NSF and the DOE National 
Laboratories have found site visits especially valuable for more 
complex research programs involving several investigators. As the 
ARS consolidates the number of ORIS units and encourages more 
interdisciplinary, collaborative research, the importance of this 
form of peer review may increase. 



Temire 

The ARS should seek legislative or administrative permission to 
employ nontenured research scientists for periods greater than one 
year. A longer time frame would enable the ARS to evaluate 
the research capabilities of these scientists and choose only the 
best as tenured ARS researchers. An excellent model is the NIH's 
Intramural Staff Fellowship program. 

Competent judgment of individual creativity takes more than 
one year. In universities, five to six years are typically required 
before a scientist is considered for tenure. Thus, a system where 
tenure decisions must be based on a single year's experience is 
counter to the goals of the ARS. 
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Adriiory Council 

The committee endorses the recommendation of the 1985 NRC 
report New Directions for Bioaeienees Research m Agriadture: 
High Reward Opportunities that a research advisory council report- 
ing to the administrator be established. An important function of 
this council could be oversight of the peer review system. 
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APPENDIXES 



A 

Research Project 
Peer Review Form (ARS-415) 



AGRICULTUItAL RCSCARCH 

RESEARCH PROJECT PEER REVIEW 



A— Projtct D«ti «nd P— r Rtvww Typg ~ To be completed by Research Leader prior to solicitation of peer review. 



PROJECT DATA 



f ROJeCT Tl TL,K: IPROJECT NO 



OATC 



REVIEW TYPE 



AR Of Non • AR 
Scltntlst 



□ 



AR Technical 
Advisor 



□ 



National Program 
Staff Coordinator 



□ 



Biometnclan 



B- 



Scttntific Mtrit Rtvitw Criteria 

To be completed by ail peer reviewers. Both qualilaiive ratings and narrative comments desired in spaces indicated; responses may be omitted on 
criteria peer reviewer does not feel qualified to address. Use additional sheets asnecessarv to nrovide re^nnn^pv 



1. Ovtrall Mttntific valut of propoMd rtMarch proi«ct: Will the proposed project make a significant 
contribution to new knowledge, provide a butter understanding of existing knowledge, develop 
appropriate new methodologies, or make valid contribution(s) to new technologies? 

NARRATIVE (PUaie provide commenti and explanation of rating)- 



(Chech one) 


Low 


below 


Average 


above 


High 













2. Adtquacy of tht rtatarch propoMi dtsign: Is the proposal adequate and scientifically feasible 
with respect to the hypothesis, approach, and plan of work ? Is the experimental design statis- 
tically sound ? 

^ O (Pleate provide explanation of rating and tuggettiom for improvement$)t 
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{Check one) 


Low 


below 


Average 


above 


High 













(Chech one) 


Low 


below 


Average 


above 


High 













3. Adequacy of li^traturt rtvitwand knowltdga: Docs ihc pro|cci siaicmeni provide an adequate 
review of the literature and demonstrate an appropriate awareness of the current state of the art' 

NARRATIVE (Please provide comments and explanation of rating). 



4. Adaquacy of methods, equlpmant and parsonntt proposed- Are the proposed methodologies, 
equipment and personnel appropriate and sufficient to accomplish the obfective ? 

NARRATIVE (Pleasc provide comments and explanation of ratinfif 



(Chech one) 


Low 


belQw 


Average 


above 


High 
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9. Appropriattnm off propottd timtfraiiM: Can the obiective(s) be reached within the timeframe of the pro- 
posal and what Is the probability of success? Can the objectives be restated so as 'o be achieved in the 
stated timeframe? 



; (PleaB€ provide exphnation of rating and iuggettions for i:nprovement$}: 



(Check one} 


Low 


Delow 


Averag« 


above 


High 













i. Scientific importance off tha propoMi and Its relationship to ongoing rosaarch: What is the degree of 
Kientific relevance and urgency of thr proposed research and will the results contribute materially to 
the success off other ongoing projects : 

NAllltATlVB (Pl9as9 provide explanation of rating and $ugge$tion» for improvement$)'. 



(Check one} 


Low 


b«iow 


Average 


above 


High 













7. ExiMit off duplicate off any (State, Federal or private) ongoing effort: D'>es the project as proposed duplicate any (State, Federal or private) ongoing research 
ipetitive off previous research ? Iff so, is this duplication/repetition desirable ? 



t. RtlaUvt proportion of iMik, appliMl and dtveloprntnttl rtwarch: In the opinion of the peer reviewer, what percentage of the proposed research to Imie in 
nature, what percentage ii apolM.in nature, and what percentage is dtvtlopmtntal In nature ? 



Progretn Merit Review Criteria 

H To be completed by NPS reviewer and by other peer reviewers who feel guaiified to address the criteria and questions. 

1. Fit cf proposed project to research priorities? 
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2. Need for research within context of National Research Programs, Technological Objectives, and USDA ARS missions and goals? 



3. Feasibility of completing the proposed research and achieving the objectives within the timeframe of the "need"? 



Q * Judgment of the cost/ben«. 'It of the proposed research ? Adequacy of the funds and personnel requirements and facilities proposed ? 
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. Adequacy of Ihc beneficiaries identified for the proposed proiect ? (A-e the clientele and beneficiaries ofthu reieorch propeHy identified r Are they 
appropriate to A RS mtiMtorts and goals, ..Mtrur 



P— r Rtvitwer Information (Will be provided to project leader unlets specified to the contrary by 



peer reviewer.] 



ADDRESS 



TITLE 



SIGNATURE 



TELEPHONE NO. 



ARS Form 415 (1/82) (Page 3) 
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The Six Major Objectives 
of the ARS Program Plan 



The ARS Program Plan states six major objectives (ARS, 
1983). The 1986 distribution of funds for the plan is *:hown in 
parentheses (ARS, 1985a). The objectives and fund allocations 
guide the mission and focus of ARS research. 

1. Maintaining and increasing the productivity and quality of 
crop plants (39%). 

2. Increasing the productivity of animals and the quality of 
animal products (19%). 

3. Acfiieving maximum use of agricultural products for do- 
mestic markets and export (19%). 

4. Managing and conserving soil and water resources (1S%). 

5. Promoting human health through improved nutrition and 
family resource management (8%). 

6. Integrating scientific kr awledge of agricultural production, 
processing, and marketing into systems that promote resource 
management and transfer of technology to users (2%). 
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Other ARS Review Systems 



ARS RESEARCH POSITION EVALUATION SYSTEM 

The ARS uses a combination of the Federal Research Grade 
Evaluation Guidelinea issued by t?ic Office of Personnel Manage- 
ment and its own related policies U> review its scientists for promo- 
tion and tenure. The ARS Research Position Evaluation System 
Handbook (ARS, 19o7) documents this system. The scientist un- 
der review writes a description of his or her duties, responsibilities, 
qualifications, and scientific achievements. The scientist's research 
leader and area director must approve this document. The ARS 
personnel division chooses a seven-person in-house review panel 
that includes one person from management, one person from per- 
sonnel, and five peer scientists to evaluate and rate the scientist. 
Final panel decisions are based on consensus and released in a writ- 
ten report. The report includes observations, recommendations, 
and a rating that establishes the scientist's grade. The scientist 
and his or ner supervisor then discuss the rating. ARS scientists, 
ARS management, and persons in other federal agencies highly 
regard the ARS Research Position Evaluation System. 

29 
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ABS PROGRAM REVIEWS 

The National Program Staff conducts formal and informal 
ARS site visits to review programs and obtain information needed 
for National Program Staff decisions on allocation of ARS re- 
sources. The National Program Staff organizes and leads the 
formal reviews. In general, the National Program Staff identifies a 
program area or specific location for review and invites participar 
tion by its national program leaders in that field of research, the 
area director, research leaders, and other appropriate managers 
and scientists from ARS. Occasionally, the National Program Staff 
asks non-ARS scientists who are emplc^ed at universities and in- 
dustries and who are experts in the field of science under review to 
participate. Specialists from agencies inside or outside the USDA 
(for example, the Soil Conservation Service or the Environmental 
Protection Agency) , who represent the user conmiunity, may also 
participate. The review team, which a national program leader 
generally chairs, writes a report based on the site visit. The Nar 
tional Program Staff headquarters, the area office, and the research 
leader under review use the review team's findings. 

ARS program reviews are primarily focused on evaluating 
the progress of research programs and as such are retrospective 
reviews. The National Program St&ff can use information from 
program reviews as one form of input to change national research 
approaches or objectives, redirect funds and personnel from one 
project or location to another, and terminate or initiate ORIS 
project work units. The National Program Staff strives to visit a 
quarter of the ARS locations each year. On the average, then, a 
particular laboratory would be visited every four or five years. 



D 

In-House Review 
in Other Federal Laboratories 



NATIONAL INSTITUTES OF HEALTH 

Each of NIH's institutes that has an intramural research pro- 
gram also has an advisory group called a Board of Scientific Coun- 
selors. The board is responsible for retrospective review of the 
particular institute's laboratories. Each board consists of 12 to 15 
senior scientists from academic institutions throughout the United 
States. A particular board's members have expertise in disciplines 
matched to the research activities of the institute they review. A 
board is often assisted by outside consultants with special exper- 
tise. 

A board reviews a given laboratory within an institute about 
once every three years. Reviews are conducted through site visits, 
during which the laboratory scientists present their studies to the 
board members. Laboratory senior scientists as well as more ju- 
nior postdoctoral fellows who are being considered for tenure are 
expected to defend their work during questioning by and discus- 
sion with the board and its consultants. The laboratory chief is 
responsible for the overall presentation of the work in the labora- 
tory. 

Based on the site visit, the board evaluates the scientific merit 
of the laboratory projects; decides whether the laboratory has an 
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appropriate amount of funds, space and personnel; recommends 
junior professional staff to be considered for tenure; and assesses 
whether the laboratory chief provides suflScient overall leadership, 
support, and advice. The board writes a report on the laboratory 
activities and personnel and makes specific reconmiendations. The 
board gives this report to the scientific director of the intramural 
program of the particular institute, the director of that institute, 
and the deputy director for NIH intifjaural programs. The board's 
reconmiendations form a basis for int8£?ial NIH reconmnendations 
regarding allocation of space, funds, and personnel slots; promo- 
tions for senior staff; and tenure plans for junior staff. 

It should be pointed out that postdoctoral fellows may serve in 
that role for no more than seven years. After that time, they must 
leave if they have not been recommended for tenure. If tenure will 
not be granted, a scientist receives a one-year notice. Only about 
one in ten postdoctoral fellows receives tenure. 



DEPARTMENT OF ENEKGY 

Programs at the DOE National Laboratories are reviewed 
in several ways to assess individual research projects and group 
efforts. Various DOE offices conduct DOE review to examine sci- 
entific programs «aid laboratories. Although different offices share 
many features of DOE review, the details of their procedures vary. 
DOE review includes laboratory site visits, prospective review of 
individual project proposals and new initiatives, and retrospective 
review of existing programs. These reviews are considered in the 
DOE's budget planning and allocation process. 

The DOE's Office of Health and Environmental Research and 
Office of Basic Energy Sciences support most of the biological 
research at DOE laboratories. These offices use similar review 
processes. Principal investigators must document their existing 
programs in yearly progress reports, which also include projections 
for coming fiscal years. These reports serve as budget requests to 
the DOE; they form the basis for the DOE's own federal budget 
request. In addition, outside site visit review teams examine the 
DOE's scientific programs and laboratories between every two and 
five years. The external review teams' reconunendations enter into 
DOE's laboratory funding decisions. 

When DOE headquarters are interested in new program ideas 
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originating within the National Laboratories, the ideas are sub- 
mitted to DOE through formal proposals. These proposals are 
also mailed to outside scientists for peer review. DOE staff an- 
alyze these reviews and decide whether or not to fund a given 
proposal. In addition, the Office of Health and Environmental Re- 
search has recently begun to use external review panels to assess 
project proposals and ongoing research programs. 

Procedures for internal review and contractor review of the 
National Laboratories' programs vary from laboratory to labo- 
ratory. These reviews focus on retrospective evaluation. For in- 
stance, at Brookhaven National Laboratory, small in-house expert 
panels conduct internal review. The panels use the results pri- 
marily to evaluate staff and guide promotion and tenure. The 
laboratory's bof .a of trustees make decisions on tenure, using 
procedures similar to those in academia. It should be noted, how- 
ever, that this laboratory is the only one within the DOE that 
grants tenure. 

External committees conduct the internal reviews at Los 
Alamos Natbnal Laboratory. They report their results to the lab- 
oratory director. These external committees examine programs 
within the laboratory's divisions for scientific content, merit, and 
productivity. Senior laboratory management uses these reviews to 
decide program suitability, direction, and stafiing. 

Visiting conmiittees conduct contractor review at Brookha- 
ven. These committees assess scientific programs within that lab- 
oratory's departments. Committees give reports to the board of 
trustees, which discusses the report recommiendations with the 
department. Los Alamos uses contractor review differently — to 
assess the laboratory's mission in a broad area, such as biology, 
over itJi many divisions. Because many different contractors spon- 
sor DOE laboratories, the protocols for contractor review vary 
widely. 



E 

In-House Review in Private Industry 



Review procedures in pharmaceutical and agrichemical com- 
pany laboratories generally involve internal dialogue and reports 
between scientists and managers at several levels, as well as exter- 
nal boards of scientific advisors and individual consultants. Writ- 
ten research proposals by company scientists are often the outcome 
of previous extended discussions with management. Consequently, 
the company is likely to put these proposals into action. Compa- 
nies evaluate individual scientists' accomplishments and contribu- 
tions to the firm; such reviews determine salary increases, pro- 
motions, or transfer. Small entrepreneurial companies rely more 
heavily on their outside scientific advisory boards to guide research 
directions and evaluate progress. Larger, established companies 
rely less heavily on external advisory boards. Rather, internal 
company scientists and managers regularly review their labora- 
tory operations and assess their research programs. 

At one representative large agrichemical and health care firm, 
each scientist submits an annual report sunmiarizing his or her 
accomplishments, publications, and patents. These reports are 
circulated within the firm to inform scientists and managers of 
technical advances. The management evaluates the firm's re- 
fiearch programs twice a year. For these reviews the scientists 
write detailed summaries of their current work and plans and 

35 



ERJC 



40 



s« 



IMPROVING RESEARCH THROVOH PEER REVIEW 



give oral presentations to their immediate sup ^rvisors. The man 
agement selecte the most significant projects for presentation to 
higher management and other departments within the firm. These 
processes help to evaluate and plan research and to disseminate 
technical advances within a large company. 

The management of a representative m^or pharmaceutical 
house uses a similar combination of written reports and oral pre- 
sentations for review of projects within its ^search laboratories. 
Those projects deemed meritorious rece' m wider review within 
the company. This firm has review systems for new project pro- 
posals and ongoing projects. Company scientists, managers, and 
outside experts from universities and government laboratories cri- 
tique proposals. The management uses these reviews in deciding 
which new projects to pursue. The management and departments 
not directly involved in the project review ongoing projects at least 
once a year. As projects move into the development phase, man- 
agers and research scientists review their progress more freouently. 
The firm's board of ad v^«iors, which is composed of outside scien- 
tists with international reputations in their disciplines, also reviews 
projects yearly. All reviews of ongoing projects are passed through 
maniai^ment to the project scientists, who must incorporate the 
reviewers' recommendations or explain why the recommendations 
are not appropriate to their research. 
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Recent PnblicationB of the Bon-d on Agrkultore 
PoUcy and Resources 

Agricultural Bio tachno logy: Strat « for National CompetitiveneM (1987), 224 pp., 

ISBN 0.309-03745-X. 
Regulating Pesticides in Food: Tne Del&ney Paradox (1987), 288 pp., ISBN 

0.309-03746-8. 

Pssticide Rcsbtance: Strategies and T^tics for Management (1986), 480 pp., ISBN 
0.309-03627-5. 

Pesticides and Groundwater Quality: Issues and Problems in Four States (1986), 136 

pp., ISBN 0-309-03676-3. 
Soil Conservation: Assessing the National Resources Inventory, Volume 1 (1986), 134 

pp., ISBN 0-309-03649-9. 
Soil Conservation: Assessing the National Reoources Inventory, Volume 2 (1986), 314 

pp., ISBN 0-309-03675-6. 
New Directions for Biosciences Research in Agriculture: High-Reward Opportunities 

(1985), 122 pp., ISBN 0-309-03542-2. 
Genetic Engineering of Plants: Agricultural Research Opportunities and Policy 

Concerns (1984), 96 pp., ISBN 0-309-03434-5. 
Impacts of Emerging Agricultural IVends on Fish and Wildlife Habitat (1982), 303 

pp., ISBN 0-309-03283-0. 

Nutrient Reqobrements of Domestic Animals Series 
and Related Titles 

Vitamin Tolerance of Animals (i987), 105 pp., ISBN 0-309-03728-X. 
Predicting Feed Intake of Food-Producing Animals (1986), 95 pp., ISBN 
0-309-03695-X. 

Nutrient Requirements of Cats, Revised Edition (1986), 87 pp., ISBN 0-309-03682-8. 
Nutrient Requirements of Dogs, Revised Edition (1985), 79 pp., ISBN 0-309-03496-5. 
Nutrient Requirements of Beef Cattle, Sixth Revised Edition (1984), 90 pp., ISBN 
0-309-0^447-7. 

Nutrient Requirements of Poultry, Eighth Revised Edition (1984), 71 pp., ISBN 
0-309-03486-8. 

Nutrient Requirements of Warmwater Fishes and Shellfishes, Revised Edition (1983), 

102 pp., ISBN 0-309-03428-0. 
Selenium in Nutrition, Revbed Edition (1983), 174 pp., ISBN 0-309-03375 6. 
Underutilised Resources as Animal FeedstufFs (1983), 253 pp., ISBN 0-309-03382-9. 
Nutrient Requirements of Mink and Foxes (1982), 72 pp., ISBN 0-309-03325-X. 
Effect of Environment on Nutrient Requirements of Domestic Animals (1981), 168 

pp., ISBN 0-309-03181-8. 
Fseding Value of Ethanol Production By-products (1981), 79 pp., ISBN 

0-309-03136-2. 

Nutrient Reqii *ements of Coldwater Fishes (1981), 72 pp., ISBN 0-^)09-03187-7. 
Nutrient Requ aments of Goats: Angora, Dairy, and Meat Goats in Temperata and 

TVopical Countries (1981), 84 pp., ISBN 0-309-03185-0. 
Nutritional Energetics of Domestic Animals (1981), 54 pp., ISBN 0-309-03127-3. 
Mineral Tolerance of Domestic Animals (1980), 577 pp., ISBN 0-309-03022-6. 

Further information, addiv.ionftl titles (prior to 1980), and prices 
arc availcule from the National Academy Press, 2101 Constitution Av- 
enue, NW, Washington, DC 20418. 
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The National Academy Press was created by the National Academy of 
Sciences to publish the reports issued by the Academy and by the 
National Academy of Engineering, the Institute of Medicine, and the 
National Research Council, all operating Mnder the charter granted to 
the National Academy of Sciences by the "ongress of the United States. 
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